Evaluation of the Power of Re-randomization Tests, with Application to Weather Modification Experiments

Computations of power of re-randomization tests by exact methods are known to be computationally exhorbitant. We introduce a much cheaper "naive" method of evaluating power and show both by simulation and analytically for a special case that it very mildly overestimates true power. We further derive a normal approximation to re-randomization distributions of linear statistics and illustrate its closeness to the true distributions. This yields analytical formulas for calculating power approximately at negligible computational cost. The proposed methods therefore eliminate the previously prohibitive cost of calculating power for re-randomization tests and make it practical to evaluate the sample sizes needed for a randomized experiment to be analyzed by re-randomization. (See also Gabriel and Hall, 1981). Its application to weather modification experiments is illustrated. *This is an extended version of a paper presented at the Third WMO Scientific Conference on Weather Modification at Clermont-Ferrand, July 21-25, 1980. 1. RE-RANDOMIZATION TESTS. Re-randomization (permutation) methods are receiving increasing attention. These methods compare the evaluation of data from a randomized experiment with analogous evaluations of the same data calculated as though other experimental plans had been chosen. The probabilities for significance testing can then be based entirely on the chance selection of the experimental plan, and need not involve any assumptions about the stochastic nature of the data (Cox and Kempthorne, 1963). This robustness to specification is of paramount importance in meteorological experiments, where units of experimentation have to be used as they occur. Tukey, Brillinger and Jones (1978) recently came out strongly in favor of using re-randomization tests for confirmatory analyses of weather modification experiments. This was further emphasized by Gabriel (1979, 1981) who discussed some advantages of re-randomization tests over classical parametric tests. Unless there is a major breakthrough in understanding weather processes, so that better stochastic models become available, re-randomization tests are likely to be used more frequently in evaluating weather modification. Examples of such usage go back to Adderley (1961), and include several more recent papers (Bradley and Scott, 1980; Dennis et al, 1975; Elliot and Brown, 1971; Gabriel and Feder, 1969; Smith et al 1977). Similar considerations are relevant in many other areas of study. One application of re-randomization tests is to experiments in which one randomly selects m out of N units to be treated, and leaves the N-m unselected units as controls. After imposing the treatment, its effect is evaluated by comparing some responses on the m treated units with the corresponding responses on the N-m controls. Mean responses may be compared,